Skip to content

Database: WAL, batched clears of deletion records, and giving free space back - #53

Open
ChakraFusion wants to merge 4 commits into
Liquid-co:dev-publicfrom
ChakraFusion:pr/database-wal
Open

ChakraFusion wants to merge 4 commits into
Liquid-co:dev-publicfrom
ChakraFusion:pr/database-wal

Conversation

@ChakraFusion

Copy link
Copy Markdown

As offered in #21. Three commits:

  1. Incremental auto-vacuum: a one-time VACUUM when the database is opened, then free pages handed back at open and hourly. On one PC, 663 MB became 380 MB.
  2. WAL with synchronous=NORMAL, and deletion records cleared in one transaction when a save is put back or files reappear. Putting a 240k-file save back had spent most of an hour creating and deleting a rollback journal per row.
  3. 500 paths per DELETE: 50,000 records in about 0.3 s, versus 10 s with one statement per path under the race detector.

Nothing copies opensave.db directly; everything goes through store.Open, so the -wal file needs no special handling.

Tests: the database uses WAL; compaction gives space back (using a table of the test's own); clearing 50,000 records is one transaction and fast.

🤖 Generated with Claude Code

@vercel

vercel Bot commented Oct 14, 2026

Copy link
Copy Markdown

@ChakraFusion is attempting to deploy a commit to the sivadaboi's projects Team on Vercel.

A member of the Team first needs to authorize it.

@ChakraFusion

Copy link
Copy Markdown
Author

Rebased onto today's dev-public. The conflict with #61 in internal/store/db.go is resolved as planned: Open uses #61's openDB, and its DSN now carries WAL and synchronous=NORMAL; compactOnOpen stays. Store tests pass, including #61's TestOpeningANewDatabaseCommitsOnce. (TestSessionNamesTheSnapshotAlreadyTaken in internal/daemon fails about 1 in 5 runs locally, on dev-public itself too, so it's not from this PR.)

@Omar-VF

Omar-VF commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Thanks, the WAL and batching changes look good. One request: could the one-time VACUUM run in the background after startup rather than inside store.Open? On a large database that's a long first start after updating, and it needs about as much free disk as the database itself. Doing it once the daemon is up keeps startup as fast as #61 made it.

ChakraFusion and others added 4 commits October 16, 2026 16:58
opensave.db had grown to 663 MB on one device, 262 MB of it pages that
deleted rows had left: SQLite never shrinks the file on its own, and
snapshots - each with a row per file of its save, a quarter of a million
for one game - are merged, pruned and replaced all the time.

The database is switched to incremental auto-vacuum, which takes one full
VACUUM when it is opened (8 seconds for that database, 663 to 380 MB).
After that its free pages are handed back a few at a time - when it is
opened and hourly - so deleting a snapshot makes the file smaller.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Putting Project Zomboid back after its folder was emptied spent most of
an hour in the database, the save untouched meanwhile: every file's
deletion record was cleared with a statement of its own, a quarter of a
million of them, and with SQLite's default rollback journal every one of
them created, flushed and deleted a journal file.

- Deletion records are cleared in one transaction per save location, on
  putting a save back and when files come back in a snapshot
  (ClearDeletedFiles): 50,000 take a third of a second.
- The database runs in WAL mode with synchronous NORMAL: writes are
  appended to one log instead of each making a journal file of its own.
  The database stays whole whatever happens; a power cut can cost only the
  last moments' writes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
One statement per path, even inside one transaction, took 10s for 50,000
records under the race detector on CI's runners, and the test's limit
failed. 500 paths to a statement: 50,000 records in about 0.3s here. The
test's limit is generous now (30s): what it guards against took minutes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Asked in review: VACUUM inside store.Open made the first start after
updating as long as rewriting the whole database, and it needs about the
database's size free on its disk.

- store.Open no longer rewrites anything. A new database gets incremental
  auto-vacuum for free: openDB asks for it first in the connection string,
  while the file is still empty.
- A database from an older build is switched over by
  Store.EnableIncrementalVacuum, which the daemon runs once, two minutes
  after it has started, then hands free pages back as before (and hourly).
  It is not attempted without the database's size free next to it; the log
  says so, and it is tried again at the next start.
- The log says when the one-time rewrite ran and how long it took.

Test: a database created without auto-vacuum is left as it is by Open,
switched over by EnableIncrementalVacuum, and not rewritten a second time.
It fails on the previous commit, where Open did the VACUUM.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@ChakraFusion

Copy link
Copy Markdown
Author

Done (c43626f): store.Open no longer runs VACUUM.

  • A new database gets incremental auto-vacuum for free: it's asked for first in the connection string, while the file is still empty.
  • A database from an older build is switched over by Store.EnableIncrementalVacuum, which the daemon runs once, two minutes after start, then hands free pages back as before (and hourly).
  • It isn't attempted without the database's size free next to it. The log says so, and it's tried again at the next start.
  • The store has one connection, so other database work waits while the rewrite runs (8 s on my 663 MB database), but no longer during startup.

Test: a database created without auto-vacuum is left alone by Open, switched over afterwards, and not rewritten a second time. It fails on the previous commit.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants